# Updating Project Requirements

Our pip workflow is based on the approach proposed in [this article](https://kennethreitz.org/essays/2016/02/25/a-better-pip-workflow).
Note that `requirements/*` lists unpinned requirements that we added to the project while `requirements.txt`
and `requirements_dev.txt` list pinned dependencies and sub-dependencies. Pinning requirements
are necessary to be able to generate reproducible environments (rather than installing by mistake a new package
that might fail to install due to a backward-incompatible change). `requirements/*` helps us track which
packages we added to the project intentionally as opposed to ones installed as sub-dependencies. It also
allows us to constrain certain packages to specific versions like limiting django to LTS versions or pinning
a dependency due to a bug or pinning report generation packages since each update requires visual inspection
of generated reports.

## Steps for updating project dependencies

We will try to have a monthly check (approximately every other sprint) during which project dependencies are updated.

At a high level, the flow is: use `pip-outdated` to identify packages that need to be updated and then use
`pip-compile` to update `requirements.txt` and `requirements_dev.txt`. The detailed steps
for doing that are the following:

- run `pip-outdated requirements/production.txt`. That should return the list of packages in `production.txt`
that have new versions. The idea is that we want the "Installed" column to be the same as the "Wanted" column.
Also, we should run `pip-outdated` with `requirements/base.txt` and `requirements/local.txt`.
- If there are packages to update, update them with
`pip-compile --output-file=requirements.txt --upgrade-package '<package>==<version>' requirements/production.txt`
`pip-compile --output-file=requirements_dev.txt --upgrade-package '<package>==<version>' requirements/local.txt`
- Do the last step for all packages that need to be updated and make sure to test the corresponding functionality
of packages that were upgraded.
- Be sure to check the changelog of the packages to understand the changes introduced in the new version
(especially breaking changes).

- choose one package and add a test for it. This helps in reducing the burden of manual testing in future upgrades.

## Steps for adding a new package

- The package should be added to either `base.txt`, `production.txt` or `local.txt`. It should be unpinned unless
there is a very specific reason on why it is pinned and in that case, add the reason as a comment in the
corresponding requirements file.
- Then, run to update the auto-generated requirements for prod and dev:
  - `pip-compile --output-file=requirements.txt requirements/production.txt`
  - `pip-compile --output-file=requirements_dev.txt requirements/local.txt`

## Steps for removing a package

This is similar to adding a package. First, remove it from `base.txt`, `production.txt` or `local.txt`. Then, run
these commands to update the auto-generated requirements for prod and dev:
- `pip-compile --output-file=requirements.txt requirements/production.txt`
- `pip-compile --output-file=requirements_dev.txt requirements/local.txt`

## Few notes around package upgrades

- You can check the docs for pip-tools to know more about how it works.

- This process updates unpinned packages in `base.txt`, `production.txt` and `local.txt`. For the ones
that are intentionally pinned, please check the reason they are pinned and see if it's possible to upgrade
them to the latest.

- For Django, we will update the patch version every other release to get security updates, but major
updates are done by moving from one LTS to the next every 2 years and is handled in its own ticket.

- For report packages, they should always be pinned since any updates to them requires visual inspection
of the generated reports.